Computing devices continue to become more ubiquitous to daily life. They take the form of computer desktops, laptop computers, tablet computers, hybrid computers (2-in-1s), e-book readers, mobile phones, smartphones, wearable computers (including smartwatches, smart glasses/headsets), global positioning system (GPS) units, enterprise digital assistants (EDAs), personal digital assistants (PDAs), game consoles, and the like. Further, computing devices are being incorporated into vehicles and equipment, such as cars, trucks, farm equipment, manufacturing equipment, building environment control (e.g., lighting, HVAC), and home and commercial appliances.
Computing devices generally consist of at least one processing element, such as a central processing unit (CPU), some form of memory, and input and output devices. The variety of computing devices and their subsequent uses necessitate a variety of interfaces and input devices. One such input device is a touch sensitive surface such as a touch screen or touch pad wherein user input is received through contact between the user's finger or an instrument such as a pen or stylus and the touch sensitive surface. Another input device is an input surface that senses gestures made by a user above the input surface. A further input device is a position detection system which detects the relative position of either touch or non-touch interactions with a non-touch physical or virtual surface. Any of these methods of input can be used generally for drawing or inputting text. The user's handwriting is interpreted using a handwriting recognition system or method.
There are many applications of handwriting recognition in portable computing devices, such as smartphones, phablets and tablets, such as is in note taking, document annotation, mathematical equation input and calculation, music symbol input, sketching and drawing, etc. Handwriting may also be input to non-portable computing devices, particularly with the increasing availability of touchscreen monitors for desktop computers and interactive whiteboards. These types of input are usually performed by the user launching a handwriting input application on the computing device which accepts and interprets, either locally in the device or remotely via a communications link of the device, handwritten input on the touch sensitive surface and displays or otherwise renders this input as so-called ‘digital ink’. Conventionally such handwriting input applications are limited in their capabilities to provide a full document creation experience to users from the text and non-text (e.g., drawings, equations), since the focus of these applications has primarily been recognition accuracy rather than document creation. That is, available applications provide recognition of handwriting and conversion of the recognized handwriting into digital content, rendered as fontified or ‘typeset ink’, with various feedback mechanisms to the user, but that is generally the extent of the interaction available for the input.
If the user desires any further interaction with the output text, such as editing the content, manipulating the layout of content, or converting or adding the notes or other annotations into a document, the recognized handwriting content generally needs to be imported or otherwise integrated into a separate document processing application. This may be done automatically though the typesetting of the recognized handwriting into suitable typeset ink of suitable format or manually through redundant typing input of the handwriting, for example. The latter manual process is inherently counter-productive and is particularly performed when the perceived accuracy of the handwriting recognition is low, or the ability of the application to preserve the layout of the original handwriting is unreliable. The former automatic process itself does not present a great problem, however as the original layout of the handwriting and the actual input handwriting itself, the digital ink, is typically discarded in the import process, the user must refer back to the original handwriting in order to ascertain the original intent. For example, the user may have emphasized certain words or passages either by annotation or decoration, or through the layout of the content itself.
Some available digital handwriting applications provide the ability to edit the digital ink. However, this is generally done through the input of particular gestures for causing some sort of control, e.g., the launching of menus or running processes. The Applicant has found that when using handwriting applications users generally are unable or do not desire to learn specific gestures that are not natural or intuitive, or to make editing selections through menus and the like. Further, the requirement for particular learned methods to provide digital ink editing limits the usability of such applications, and digital handwriting more generally, as all users must learn the necessary behaviors for digital ink interaction.
Accordingly, in sharing or collaborative uses in which documents are created, edited, revised, reviewed, etc. by multiple parties, in particular parties from different areas or disciplines within enterprise organizations, for example, the ability of different ‘team’ members to interact with a digital document using handwriting varies such that the productivity possible is limited. For example, a member of upper management in the organization hierarchy may have limited experience with digital handwriting and therefore may not interact with documents using handwriting. Similar problems occur with non-handwriting tools conventionally used with document processing applications such as hardware tools, e.g., keyboard and mouse, and/or software tools, e.g., user interface (UI) tools and menus.
Further, the need to use specific UI and/or hardware tools for editing or other interactions with content created using digital handwriting, may cause disruption to the creative flow of users. As such, the manner of interaction with the content should be easy to use, intuitive and non-disruptive to the input of the content itself.